Unsupervised Learning Capstone¶
The aim of this capstone is to cluster our data set with unsupervised clustering methods and alternative models. For this purpose we used a data from a web site Turo.com, the biggest car sharing platform in the USA. There are two problems that required unsupervised learning methods:
- We need to cluster our data by location
- Categorize all cars by their characteristics. This information will help to continue the supervised learning and to create the predictive models.
Location clustering For the purpose of location clustering we used the Latitude and Longtitude assosiated with every listing in the data. We approached our dataset with k-means, dbscan and hierarhical clustering techniques. We will use the siloughette score for each of the methodics to define how precise is every method.
Car category clustering For this purpose we manually assigned categories for each car. We will try to do clustering with unsupervised clustering techniques ( K-means, Hierarchical Clustering and DBSCAN) with dimensionality reduction such as PSA and UMAP but more likely the best way will be to procees with the supervised classifier model. We will use neural network multi layer perceptron model and random forest to see what performs better.
#here is how our data looking
df.info()
Basic data exploration¶
At this stage we will check the sanity of the data, make observations on the data type, missing values, shape of the data
# Remove all columns between column index 70 to 99
df.drop(df.iloc[:, 69:100], inplace = True, axis = 1)
#replace all the missing data with 0 (we'll be checking later if we need to adjust this 0 for some variables)
df = df.fillna(0)
#mapping our location distribution
import matplotlib.pyplot as plt
plt.scatter(x=df['Longitude'], y=df['Latitude'])
plt.show()
#drop all non-active car listings
df = df[df['Status'] == "Active"]
#Let's scale the variables
scaler = StandardScaler()
df[['Latitude']] = scaler.fit_transform(df[['Latitude']])
df[['Longitude']] = scaler.fit_transform(df[['Longitude']])
For car clusterization purposes we're going to use these variables:
- 'Price per day'
- 'Year'
- 'Number of seats'
- 'Number of doors'
- 'Convertible'
- 'Fuel Type'
- 'Make'
- 'highValueVehicle'
- 'Transmission'
- "MinAge"
Our target pre-labeled variable is
- "Category"
Introducing Category variable¶
#combining make and model into category name
df1 = pd.read_csv ('Category_DB.csv', encoding='ISO-8859–1')
df['for category']= df['Make']+df['Model']
df = pd.merge (df, df1[['Car ID', 'Category']], on = 'Car ID', how="left")
Clean the variables if needed¶
#adjust some text errors
df.Make = df.Make.str.lower()
df.Model = df.Model.str.lower()
df.Category = df.Category.astype('category')
df.Category = df.Category.str.lower()
df = df[df['Category'].notnull()]
df.Category = df.Category.astype('category')
df['Category_code'] = df['Category'].cat.codes
df.groupby("Category")["Category"].count()
#Here is the categorized distribution of the variable:
sns.countplot(x=df['Category_code']);
#Here is the categorized distribution of the variable:
sns.countplot(x=df['Number of seats']);
sns.countplot(x=df['Fuel Type']);
sns.countplot(x=df['Number of doors']);
sns.countplot(x=df['Convertible']);
sns.countplot(x=df['minimumAgeInYearsToRent']);
Let's do some work for null variables:
df["Number of seats"].fillna(df["Number of seats"].mean(), inplace=True)
df["Number of doors"].fillna(df["Number of doors"].mean(), inplace=True)
df['Fuel Type'] = df['Fuel Type'].astype(str)
df['Fuel Type'] = df['Fuel Type'].apply( lambda x: x.replace ('0', 'Gas'))
Potential outliers¶
#Checking pricing by categories
sns.set(style="ticks", palette="pastel")
# Draw a nested boxplot to show bills by day and time
chart = sns.boxplot(x="Category_code", y="Price per day", palette=["m", "g"],
data=df)
CLUSTERIZATION¶
k-means clustering for location¶
The first problem that we're solving is to be able to separate our listings by location. We would like to use k-means because our data has only two variables that are measurements of distance. K-means should be the best fir for it however we're going to validate with other methods as well.
# Variable with the Longitude and Latitude
Z = df[['Car ID','Latitude','Longitude']]
K_clusters = range(1,20)
kmeans = [KMeans(n_clusters=i) for i in K_clusters]
Y_axis = df[['Latitude']]
X_axis = df[['Longitude']]
score = [kmeans[i].fit(Y_axis).score(Y_axis) for i in range(len(kmeans))]
# Visualize the amount of clusters
plt.plot(K_clusters, score)
plt.xlabel('Number of Clusters')
plt.ylabel('Score')
plt.title('Elbow Curve')
plt.show()
#Let's define labels for location
kmeans = KMeans(n_clusters = 12, init ='k-means++')
kmeans.fit(Z[Z.columns[1:3]]) # Compute k-means clustering.
Z['cluster_label'] = kmeans.fit_predict(Z[Z.columns[1:3]])
centers = kmeans.cluster_centers_ # Coordinates of cluster centers.
labels = kmeans.predict(Z[Z.columns[1:3]]) # Labels of each point
#put them on the plot
Z.plot.scatter(x = 'Latitude', y = 'Longitude', c=labels, s=50, cmap='viridis')
plt.scatter(centers[:, 0], centers[:, 1], c='black', s=200, alpha=0.5)
#This is our silhouette score for k-means
print(metrics.silhouette_score(Z[Z.columns[1:3]], labels, metric='euclidean'))
DBSCAN Clustering for Location¶
# Defining the agglomerative clustering
dbscan_cluster_l = DBSCAN(eps=1, min_samples=5)
# Fit model
clusters = dbscan_cluster_l.fit_predict(Z[Z.columns[1:3]])
print(metrics.silhouette_score(Z[Z.columns[1:3]], clusters, metric='euclidean'))
Z.plot.scatter(x = 'Latitude', y = 'Longitude', c = clusters, s=50, cmap='viridis')
plt.scatter(centers[:, 0], centers[:, 1], c='black', s=200, alpha=0.5)
This is definitely won't be our choise based on the silhougtette score
Hierarchical clustering for location¶
from sklearn.cluster import AgglomerativeClustering
# Defining the agglomerative clustering
agg_cluster = AgglomerativeClustering(linkage='complete',
affinity='cosine',
n_clusters=5)
# Fit model
clusters_h = agg_cluster.fit_predict(Z[Z.columns[1:3]])
Z.plot.scatter(x = 'Latitude', y = 'Longitude', c = clusters_h, s=50, cmap='viridis')
plt.scatter(centers[:, 0], centers[:, 1], c='black', s=200, alpha=0.5)
print(metrics.silhouette_score(Z[Z.columns[1:3]], clusters_h, metric='euclidean'))
Performance of location clustering algorithm¶
We found that k-means work the best way for the location clustering. The siloughette score is 0.65. Let's review the clusters:
### Adding location cluster into the dataframe and will do a visual inspetion
df = df.merge(Z, left_on='Car ID', right_on='Car ID')
sns.countplot(x=labels);
df_location = df[['Car ID', 'City', 'Latitude_y', 'Longitude_y', 'State', 'cluster_label_y']]
df_location.head(20)
#TOP 10 state and cities by quantity of cars listed
df.groupby(['State', 'City']).size().sort_values(ascending=False).head(10)
WORK ON CLASTERISATION AND DIMENSIONALITY REDUCTION¶
Let's try to solve the problem of car segmentations based on their parameters with unsupervised clustering techniques.
Trsnsform non-numerical variables into dummies¶
df = pd.concat([df,pd.get_dummies(df.Convertible, prefix="Conv", drop_first=True)], axis=1)
df = pd.concat([df,pd.get_dummies(df['Fuel Type'], prefix="fuel", drop_first=True)], axis=1)
df = pd.concat([df,pd.get_dummies(df['Make'], prefix="Make", drop_first=True)], axis=1)
df = pd.concat([df,pd.get_dummies(df['highValueVehicle'], prefix="value", drop_first=True)], axis=1)
df = pd.concat([df,pd.get_dummies(df['Transmission'], prefix="Trans", drop_first=True)], axis=1)
dummy_column_names = list(pd.get_dummies(df.Convertible, prefix="Conv", drop_first=True).columns)
dummy_column_names = dummy_column_names + list(pd.get_dummies(df['Fuel Type'], prefix="fuel", drop_first=True).columns)
dummy_column_names = dummy_column_names + list(pd.get_dummies(df['Make'], prefix="Make", drop_first=True).columns)
dummy_column_names = dummy_column_names + list(pd.get_dummies(df['highValueVehicle'], prefix="value", drop_first=True).columns)
dummy_column_names = dummy_column_names + list(pd.get_dummies(df['Transmission'], prefix="Trans", drop_first=True).columns)
Assign incoming and outcoming variables¶
X = df[['Price per day', 'Year', 'Number of seats', 'Number of doors']+dummy_column_names]
#Let's keep one more set for the variable without price
X1 = df[['Year', 'Number of seats', 'Number of doors']+dummy_column_names]
Y = df["Category_code"]
#Let's see how categories looking
Y.value_counts()/len(Y)
# Reduce it to two components for visualization and check how long will it take
X_pca = PCA(2).fit_transform(X_std)
time_start = time.time()
X_pca = PCA(2).fit_transform(X_std)
print('PSA done! Time elapsed: {} seconds'.format(time.time()-time_start))
UMAP¶
time_start = time.time()
umap_results = umap.UMAP(n_neighbors=5,
min_dist=0.3,
metric='correlation').fit_transform(X_std)
print('UMAP done! Time elapsed: {} seconds'.format(time.time()-time_start))
#we see that the UMAP technique is much longer and let's see how it's looking
plt.figure(figsize=(10,5))
plt.scatter(umap_results[:, 0], umap_results[:, 1])
plt.xticks([])
plt.yticks([])
plt.axis('off')
plt.show()
This isn't a good perfomace from a clustering perspective plus the calculation time is too high
umap_results7 = umap.UMAP(n_neighbors=7,
min_dist=0.3,
metric='correlation').fit_transform(X_std)
plt.figure(figsize=(10,5))
plt.scatter(umap_results7[:, 0], umap_results7[:, 1])
plt.xticks([])
plt.yticks([])
plt.axis('off')
plt.show()
6.1. K-Means¶
# Calculate predicted values.
y_pred = KMeans(n_clusters=9, random_state=123).fit_predict(X_std)
# Plot the solution.
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y_pred)
plt.show()
K-means Evaluation¶
labels = KMeans(n_clusters=7, random_state=123).fit_predict(X_std)
print(metrics.silhouette_score(X_pca, labels, metric='euclidean'))
DBSCAN (psa)¶
from sklearn.cluster import DBSCAN
from sklearn.decomposition import PCA
scaler = StandardScaler()
X_std = scaler.fit_transform(X)
X_pca = PCA(2).fit_transform(X_std)
# Defining the agglomerative clustering
dbscan_cluster = DBSCAN(eps=1, min_samples=5)
# Fit model
clusters_db_pc = dbscan_cluster.fit_predict(X_std)
#plot the results
pca = PCA(n_components=2).fit_transform(X_std)
plt.figure(figsize=(10,5))
colours = 'rbg'
for i in range(pca.shape[0]):
plt.text(pca[i, 0], pca[i, 1], str(clusters_db_pc[i]),
fontdict={'weight': 'bold', 'size': 50}
)
plt.xticks([])
plt.yticks([])
plt.axis('off')
plt.show()
print(metrics.silhouette_score(X_std, clusters_db_pc, metric='euclidean'))
DBSCAN (umap)¶
#The score is much better. Let's try with UMAP dimensinality reduction
# Defining the agglomerative clustering
dbscan_cluster = DBSCAN(eps=1, min_samples=5)
# Fit model
clusters_umap = dbscan_cluster.fit_predict(umap_results7)
print(metrics.silhouette_score(X_std, clusters_umap, metric='euclidean'))
Hierarchical clastering (psa)¶
scaler = StandardScaler()
X_std = scaler.fit_transform(X)
Xpca = PCA(2).fit_transform(X_std)
from sklearn.cluster import AgglomerativeClustering
# Defining the agglomerative clustering
agg_cluster = AgglomerativeClustering(linkage='complete',
affinity='cosine',
n_clusters=3)
# Fit model
clusters_h = agg_cluster.fit_predict(X_pca)
pca = PCA(n_components=2).fit_transform(X_std)
plt.figure(figsize=(10,5))
colours = 'rbg'
for i in range(pca.shape[0]):
plt.text(pca[i, 0], pca[i, 1], str(clusters_h[i]),
fontdict={'weight': 'bold', 'size': 50}
)
plt.xticks([])
plt.yticks([])
plt.axis('off')
plt.show()
print(metrics.silhouette_score(X_std, clusters_h, metric='euclidean'))
Hierarchical clastering (umap)¶
print(metrics.silhouette_score(X_std, clusters_h_umap, metric='euclidean'))
the rate is very low, we will be skipping it.
UNSUPERVISED CLUSTERING. CONCLUSION¶
During unsupervised clustering process we were unable to define clear unsupervised clustering solution for our problem. However, based on the silhouette score the best result is coming from DBSCAN clasterisation using pca technique was high we still can't rely on it. We need to go with more sophisticated approach since our variables contain mixed (numeric and non-numeric and categorical) types of data. Also, there are some sort of neural charachteristics for the clusters. In this case we need to move forward and try to create a classificator with neural network to be able to assign the category for all new cars that will be added into the system.
Multi Layer Perceptron model¶
The power of neural networks come from their ability to learn the representation in the training data and how to best relate it to the output variable that you want to predict. We will try to use MLP model as well as Random Forest Classifier and compare the performance
# Import the model.
from sklearn.neural_network import MLPClassifier
# Establish and fit the model, with a single, 1000 perceptron layer.
mlp = MLPClassifier(hidden_layer_sizes=(1000,))
mlp.fit(X, Y)
mlp.score(X, Y)
from sklearn.model_selection import cross_val_score
cross_val_score(mlp, X, Y, cv=5)
Let's try it with the standartized X values - X_std
# Import the model.
from sklearn.neural_network import MLPClassifier
# Establish and fit the model, with a single, 1000 perceptron layer.
mlp = MLPClassifier(hidden_layer_sizes=(1000,))
mlp.fit(X_std, Y)
mlp.score(X_std, Y)
This is definitely much better. We've got our model explained by 10points better when our data is standartized
from sklearn.model_selection import cross_val_score
cross_val_score(mlp, X_std, Y, cv=5)
This is a solid result for the data on average 0.69¶
Let's try another clasifier to see if our results can go higher
Random forest classifier¶
from sklearn import ensemble
from sklearn.model_selection import cross_val_score
rfc = ensemble.RandomForestClassifier()
cross_val_score(rfc, X, Y, cv=5)
these numbers are good but the best performance we've got with the
CONCLUSIONS¶
In this capstone we tried to solve the problem of clustering the data. Our original raw data has variables that should be clustered in order to make more precise predictions and to reduce the amount of noise and overfitting. Unsupervised clustering is mainly a process of putting several objects into one bin while models is an ability to predict an outcome. Two issues were solved:
- clustering listings by location
- clustering by car characteristics (price, make, type of fuel, high/not high value of the car
1. Location clustering: The unsupervised learning worked well for clustering numerical information with Lat and Long (we got the silhouette score on the level of 0.65 with k-means. We made a visual inspection and confirmed that the results are relevant and we can use them in futhure development. K-means is a valuable tool for location clustering itself and we saw it comparing its result with other methods.
2. Car category clustering we tried several unsupervised learning techniues but non of them work good for hybrid variables. We were able to achieve pretty high silghouette score with dbscan but still it's not something that can be useful. K-means and hierarchical clustering are good at finding circular (or convex) clusters, which makes them great tools for identifying well-separated clusters. But, unfortunately, they're not good at identifying clusters that are not well-separated or that have non-convex shapes such as rings inside rings. This problem better should be solved with the supervised classifier model since there are more mixed data involved that has non-linear connections. I put at the report these next steps since it's important to present how high score we got with the help of neural networks. We used supervised neural network classifier that gave us around 0.70 score. We check models with cross validation score to make sure there is no overfitting in the data.
These findings are extremelly useful for the development an investment tool in order to predict potential earnings from sharing cars on the platform. We're going to continue our work on this data set in order to increase the relaibality of our calculations.